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Abstract: This paper describes the theoretical basis and development of the iSELF: an Internet-tool for Self-Evaluation and 
Learner Feedback to stimulate self-directed learning in ubiquitous learning environments. In ubiquitous learning, learners 
follow their own trails of interest, scaffolded by coaches, peers and tools for thinking and learning. Ubiquitous learning 
solutions include on- and off-line, formal and informal learning. To benefit from its possibilities, learners need to develop 
competencies for self-directed learning. To do so, a self-evaluation tool can help the learner to get insight in his/her own 
development, to manage and monitor his/her own learning process, to collaborate in learning, to relate the learning to 'real 
life' needs, and to take control over educational decisions. The iSELF was developed in an iterative process, complying to 
the following high level requirements: (1) Enabling learning anytime, anywhere; (2) Supporting self-directed learning; (3) 
Evaluating learner, learning solutions and job-needs; (4) Assessing learner competencies; (5) Using card-sort method for 
questionnaires; (6) Facilitating questionnaires 'under construction'; and (7) User-friendly design. The resulting online tool 
contained a card-sort module, looking somewhat like a 'solitaire' game, a profile module to evaluate core competencies, 
and a feedback module to suggest learning possibilities. For illustration, 14 different studies that contributed to the 
development of iSELF and to the development of self-evaluation questionnaires compliant to iSELF, are briefly discussed. 
These illustrative studies included various populations: e.g. students, employees from small and medium enterprises, crisis 
management organizations, and the military. Usefulness and usability of the self-evaluation tool were valued positively. The 
iSELF contributes to an adaptive ubiquitous learning environment in which the learner can make the educational decisions 
according to self-directed learning principles. The iSELF will stimulate self-directed learning in a ubiquitous learning 
environment and will help to create learners for life. 
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1. Introduction 

1.1 Ubiquitous learning needs self-directed learners 

Nowadays, technology is very much part of everyday life and work (Mork 2011). Information and knowledge is 
handled and shared by using ubiquitous technology; ICT that makes it possible to access information 'anytime, 
anywhere' (Adkins et al. 2002). A learning environment that makes use of this technology is often referred to 
as 'ubiquitous learning'. This is a way of learning in which learners follow their own trails of interest, scaffolded 
by coaches, peers and tools for thinking and learning (Dieterle & Dede 2007). Ubiquitous learning solutions 
include on- and off-line, formal and informal learning. 

The availability of ubiquitous learning possibilities may assume that learners are able to learn and will develop 
themselves anytime, anywhere (G. D. Chen et al. 2008). However, this assumption might be too ambitious. 
Since the ability to manage one's own learning is becoming increasingly important, one of the goals of 
education should be to create learners for life. Learners for life can be described as learners who have a flexible 
and pro-active attitude toward learning and developing themselves (Du Bois & Staley 1997). In this context the 
concept of self-directed learning is often mentioned and intensively discussed (Collins 2004). 

In a review study of Stubbe &Theunissen (2008), five crucial elements of self-directed learning were identified: 

■ Learner control: Control over educational decisions and learning process. 

■ Self-regulating learning strategies: Skills that support the learner to manage and monitor his/her own 
learning process (e.g. setting goals, planning, problem solving, and strategy use). 

■ Reflection: The combination of self-assessment and self-evaluation on both the performance and the 
learning process that gives the learner insight in his/her own development. 
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■ Interaction with the social environment: The interaction with others, learners and teachers/coaches, in 
order to determine what goal should be set, discuss in what way this goal can be achieved, cooperate and 
collaborate during the learning process and ask for help. 

■ Interaction with the physical environment: The learning experience should be set in the 'real world' and 
should relate to 'real-life' (work)situations. 

Reality shows that some people develop a self-directed attitude toward learning, especially in relation to work 
or a hobby, others do not (Collins 2004). Explicitly teaching self-regulating learning strategies or reflection and 
stimulating (perceived) learner control, helps learners to become more self-directed (Stubbe & Theunissen 
2008). Therefore, a ubiquitous learning system must not only provide the learner with learning resources 
anytime and anyplace (C.-M. Chen & Li 2010). It must also actively provide the learner with the appropriate 
learning possibilities for self-directed learning (Hiemstra 2006; S. L. Wang & Wu 2011). Self-directed learners 
are not merely consumers of learning facilities, they should also be able to contribute to the facilities by 
sharing knowledge and supporting other learners (Koper et al. 2005). 

Self-evaluation (also called 'self-monitoring 1 or 'self-assessment') refers to an individual systematically 
observing his/her own behaviour and performance. Realistic self-evaluation will help the learner to gain self- 
regulatory control (Boekaerts 1991). Therefore, to become a self-directed learner, one should get insight in 
one's own development. During self-evaluation, the learner makes a comparison between the noted behaviour 
and some designated standard (Hughes et al. 1991). In order to be able to assess his/her performance 
realistically, the learner must have access to internal standards of performance, involving a definition of what is 
meant by a 'good' or 'adequate performance. A person acquires these internal standards and self¬ 
conceptualizations on the basis of his experiences but also on the basis of statements made by significant 
others and self-attributions (Boekaerts 1991). Therefore, peer or expert observation can help learners to 
evaluate their own opinion about themselves. Self-evaluation as motivator for future behaviour agrees with 
Bandura's Social cognitive theory of self-regulation, assuming that a person can become can become 
knowledgeable about his/her own capabilities and skills (Bandura 1991). 

Although self-evaluation presumes higher order cognitive skills, it is proven to be possible even in populations 
of people with mild cognitive retardation (Hughes et al. 1991). There are indications that self-reported abilities 
and competencies have concurrent validity with ratings by others, although it is found that self-assessment can 
be somewhat more positive (Jones & Fletcher 2004; Kelso et al. 1977). Moreover, 'People may not be right 
about themselves, but their self-evaluations are the ones that most powerfully affect their future behaviour' 
(page 45, (Byrnes 1984). As a result, self-evaluations are relevant for learner behaviour. 

Thus, self-evaluation will help the learner to get insight in his/her own development, to manage and monitor 
his/her own learning process, to collaborate in learning, to relate the learning to 'real life' (work)needs, and to 
take control over educational decisions. In this way, all five elements of self-directed learning, mentioned 
before, are stimulated by self-evaluation. 

2. Assessment of competencies 

To answer the question: 'What to evaluate?' we specifically look at the fifth element of self-directed learning: 
Interaction with the physical environment. Learning needs to be related to 'real-life' (work)situations, because 
meaningful knowledge is constructed only when process of learning integrates with cultural and life contexts 
(C.-M. Chen & Li 2010). In our rapidly changing society, initial training alone cannot meet the need for the 
development of working individuals. Training results become obsolete the moment they are obtained. A 
flexible and innovative economy requires permanent adaptations of knowledge, skills and attitudes, also called 
'competencies'. Competencies are indivisible clusters of skills, knowledge, conduct, attributes and notions (e.g. 
(Boekaerts 1991). They are context dependent, connected to activities and tasks, but also flexible in time (van 
Merrienboer et al. 2002). In their essence, competencies can be used in more situations than the current task. 
This means that when one's job changes, the acquired specific skills become obsolete, but the acquired 
competencies can still be useful. Nevertheless, in our fast changing society it is possible that competencies 
themselves become obsolete or less important. 

Another characteristic of competencies is that they can be acquired by learning and development. 
Competencies can be valuable to match individual performance and career planning with organizational job 
needs (Marko & Savickas 1998). In that context the concept 'core competencies' is used, competencies which 
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are essential for certain tasks or positions (Case 2003) and as such will provide the content for the relationship 
with 'real-life' work situations. 

Reporting on one's behaviour poses a difficult cognitive task and participants' reports are influenced by the 
wording of questions, format, and content (Schwarz & Oyserman 2001). In a study on self-assessment for 
selection purposes it was found that measurement conditions have substantial positive impact on the quality 
of self-ratings (Jones & Fletcher 2004). Self-ratings appeared to improve for instance when social comparison 
instructions were given; when they expected external validation; the anonymity of raters, previous self- 
assessment experience, unbalanced, positively toned scale; motivational instructions; framing questions 
unambiguously, ensuring measures are tied to actual performance, specifying the time period under 
consideration (i.e., past, present, or future behaviour); competencies broken down into distinct dimensions. 
However, it was also found that individual and gender differences were substantial and should be taken into 
account (Jones & Fletcher 2004). A self-evaluation instrument in a ubiquitous learning environment needs an 
easy to use, flexible and reliable method to gather information on selected competencies. A card-sort method 
is such a method, with good psychometric characteristics (Lievens & Sanchez 2007). Card-sorts involve the 
placement of cards onto piles, based on how each participant feels the concepts or statements on them are 
related. When using this method for evaluation of competencies, competency statements can be placed on the 
cards. Former research with this technique showed that people are able to sort a large number of separate 
cards in a relatively short time, which will increase learners' motivation to use it for evaluation. The technique 
is particularly useful for identifying the common ground between a larger and diverse collection of 
competencies with a large and diverse group of participants (Caldwell & O'Reilly 1990). 

There are two approaches in card-sort: the free and the restricted procedure (Harper et al. 2003). In the free 
approach, a participant is allowed to make as many piles of related cards as necessary, and label them. In the 
restricted approach, a participant uses piles that have already been defined (e.g. questionnaire Likert scales like 
'not applicable at all' to 'totally applicable'). This enables the use of statistical techniques to cluster related 
competency statements into core competencies. The restricted card-sort relies less on the categorization skills 
of the participants. This makes it useful for a sample with various levels of education and experience, as is the 
case in a ubiquitous learning environment. Moreover, a standardized categorization makes it possible to make 
learner profiles that can be related to peers and that can show development over time. The same 
categorization can also be used to evaluate job-needs or the available learning solutions. An automatic match 
of the learner profiles with the learning solution profiles will show if they are beneficial to the learner. The 
results should be presented as suggestions so that the learner can make the educational decisions according to 
self-directed learning principles. 

The aim of this paper is to describe the development of the iSELF: an Internet-tool for Self-Evaluation and 
Learner Feedback to stimulate self-directed learning in a ubiquitous learning environment. The theoretical 
background and characteristics of the iSELF, based on the developments so far, will be discussed. In addition, 
the first experiences with the tool and the matching self-evaluation questionnaires will be illustrated on the 
basis of 14 different studies. 

3. The iSELF itself 

3.1 High level requirements 

With the introduction text in mind, a set of requirements was developed for the iSELF: 

1. Enabling learning anytime, anywhere: available for every learner through internet, with the possibility to 
embed the tool into Learning Management Systems. 

2. Supporting self-directed learning: helping the learner to get insight in his/her own development, to 
manage and monitor his/her own learning process, to collaborate in learning, to relate the learning to 'real 
life' (work)needs, and to take control over educational decisions. To support control and to provide a 'safe' 
learning environment, it is important that the learner is the only one who can see personal evaluation 
results, until he/she decides otherwise. 

3. Evaluating learner, learning solutions and job-needs: possibility of using the same content: (a) to score the 
learner competencies of all kinds of learners (e.g. low and high educated), (b) to assess the learner 
competencies by the learners themselves or by peers, colleagues, coaches or subject matter experts, 
invited by the learner, (c) to score which competencies are trained by certain learning solutions or are 
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relevant for a job or position. Using the same content in all situations enables a comparison between 
them. 

4. Assessing learner competencies: assessing competencies that are specific for a group of learners in their 
(work)situations, and at the same time generic enough to remain relevant in our rapidly changing society. 

5. Using card-sort method for guestionnaires: this technique is less time-consuming and more objective than 
other methods. This will increase motivation to use it in a large and diverse group of participants. 

6. Facilitating guestionnaires 'under construction': with new developments in the workplace, new 
competencies will become important. Therefore, new questionnaires will be developed all the time. 

7. User-friendly design: most people do not like questionnaires. A playful appearance, user-friendly operation 
and clear, relevant content will increase motivation. 

3.2 Design 

The overall structure of the iSELF is presented in Figure 1 and explained in the next paragraphs. 


Card-sort 


ik A A A 

Learner Invitee Educational Subject matter 

expert expert 



Analysis 1 


i 


l 


l 



A 

Administrator 


Figure 1: Overall structure 

3.2.1 Users 

The overall structure includes the following iSELF users: 

1. Learner: the learner in a ubiquitous learning environment. 

2. Invitees: peers, colleagues, coaches or subject matter experts, invited by the learner to assess the learner. 

3. Educational expert: the evaluator of available learning solutions. 

4. Subject matter expert: the evaluator of competency requirements for a job or position. 

5. Administrator: administrates the tool content: competency statements, information about core 
competence clusters and reference group- or norm figures. 

3.2.2 Card-sort 

The appearance of the card-sort tool, a front-end input module, is somewhat like a 'solitaire' game (see Figure 
2). Instead of sorting playing cards, learners sort competency statement cards on their importance. The tool 
offers the possibility to get an overview or skip a statement temporarily. The competency statements are 
formulated in such a way that they are easy to comprehend for people with different backgrounds. A second 
person singular verb at the beginning of every statement emphasises the fact that competencies express 
themselves in behaviour (E.g. 'Cooperates with people from other organizations' or 'Uses ICT systems to collect 
information and knowledge quickly'.) These statements can be used with different overall opening questions. 
For the learner the opening question could be: 'In the last two weeks, was this applicable to you?' For the 
invitee it would be: 'In the last two weeks, was this applicable to [name learner]?' For the use with the learning 
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solutions of job-requirements: 'Is this applicable to [name learning solutions or job]?' The card-sort module 
was built using Adobe Flex, an open source framework for building rich Internet applications that are delivered 
via the Flash Player 6.0. 


^ u 1 ^ 0 
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Figure 2: Card-sort module in input mode 
3.2.3 Profiles 

The profile module shows the results of the card-sort input to the learner. After analyses, the competency 
statements are clustered into core competencies. The results are presented to the learner in one or more 
graphs. It can show learner results in combination with reference scores or scores from former sessions. The 
profile module was built using ColdFusion 8.0, a rapid application development platform that includes 
advanced features for enterprise integration and enables the development of rich Internet applications. 

3.2.1 Feedback: suggestions for further learning 

The feedback module can be used to compare perspectives by interpreting the results of both the learner and 
the invitee. The feedback can also be used to provide suggestion for learning solutions beneficial to the learner 
or for job-training and selection. The feedback is based on an automatic match of the learner scores with the 
learning solution scores. The feedback module was built using ColdFusion 8.0. 

3.2.2 Administration 

The administration module, a back-end module, combines several input functions: overall opening question, 
statements, number and content of answering categories, assignment of statements to clusters, personal 
accounts, data export, reference or norm score input. The module was built using ColdFusion 8.0 running Railo 
Server, the main version of Railo (a compiler) which can be integrated into standard web servers. It is suitable 
for production use. 
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Figure 3: A learner profile 
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Figure 4: Administration of competencies 


3.2.3 Content: psychometric sound questionnaire 

The content of the iSELF card sort module is created according to social sciences standards in questionnaire 
construction (Schwarz & Oyserman 2001)D. Currently, four questionnaires are under development (see Table 
1). The presented scales are based on theoretical concepts and data analyses so far (more information can be 
obtained from the authors). At any moment in the development of a questionnaire it is possible to export the 
data for statistical analyses for validation and reliability tests. Using a.o. principal component analysis (factor 
analyses) and Cronbach's alphas (> 0.7), clusters can be identified and transformed into scale scores. The 
results from these analyses can be fed back through the administration module. The data export option can 
also be used to perform additional statistical analyses for group comparisons and determination of the 
influence of background variables like age or experience. These possibilities make the iSELF not only a learner 
tool but a scientific tool as well. 
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Table 1. Questionnaires developed in iSELF-style 


Questionnaire 

No. Scales 

Scales 

items 

Example of item 

Competencies for 
deploying 
simulators in 

Military training 
(ComSim) 

5 

Using simulators, Coaching, 
Advising, Teaching and 

Organizing, Accurate planning 
and documentation. 

65 

'Translates learning goals into the 
possibilities of a certain simulator 1 

Competencies for 
multidisciplinary 
cooperation in a 
Network Centric 
Organization 
(NCOQ) 

7 

Leadership skills, Working in ad 
hoc teams, Open mind for ICT 
tools, Own role in behalf of the 
team, Informatin processing, 
Social skills, Communication 
skills. 

71 

'Cooperates with people from 
other organizations' 

Competencies for 

Self-directed 

Learning 

Questionnaire 

(CSLQ) 

5 

Learner control, Self-regulating 
learning strategies, Reflection 
on learning and performance, 
Collaborative learning, 

Reflection on relevance for 

work. 

32 

'Chooses what to improve in your 
work' 

Military 

leadership 

competencies 

(MilLead) 

8 

Communication, Take initiative, 
People oriented, Employee 
development, Organization 

orientation, Planning and 
organizing; Results-oriented, 
Vision 

40 

'Takes the personal situation and 
wishes of others into account' 


4. The iSELF: development in phases 

The development of the iSELF was an iterative process in which the modules (card-sort, profile and feedback) 
were prototyped, built and tried successively. Experiences were gathered using 14 different studies with 
various study aims and designs. All studies contributed to the development of the iSELF and will briefly be 
discussed as illustration of the iSELF possibilities. The general characteristics of the studies are presented in 
Table 2. More information about these studies can be obtained from the authors. 

4.1 Phase 0: Prototyping with a paper card-sort tool 

4.1.1 The paper-based iSELF 

In this paper-based version, the iSELF only contained the card-sort module. The selected competency 
statements were each placed on a paper cards. Together with this set of cards came five envelopes with the 
labels 'not important', 'somewhat important', 'important', 'very important' and 'essential'. A participant could 
express the importance of a competency statement for a position by assigning the cards to the envelopes. In 
this way, a 5-point Likert-scale was created for each statement. 

4.1.2 Context 

The paper-based prototype was used for career planning using core competencies (see Table 2, study 1). Semi- 
structured interviews and a document study resulted in 65 cards with competency statements. Subject matter 
experts sorted the cards twice for their own function: on expert and on novice level. In a group discussion both 
the method and the content of the cards was discussed. Using principal component analysis, five core 
competencies could be identified. 
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4.1.3 Experiences 

The participant sorted the large set of statements cards quickly: each round of 65 cards took about 5 to 15 
minutes. Most participants were positive about the procedure and preferred it to a standard questionnaire. 
The card-sort module was considered useful. 


Table 2. General characteristics of the studies with iSELF 


Refr. 

nr. 

iSELF modules 
used 

Questionnaire (1) 

N 

Domain 

Subjects 

characteristics 

Study aim 

Research type 
(2) 

i 

Prototype 

Card-sort 

ComSiM 

28 

military 

Subject matter 

experts training- 

related functions in 
the Royal 

Netherlands Army 

Identification of 

competencies for career 
planning 

obs 

2 

Card-sort 

NCOQ 

158 

crisis 

management 

organizations 

Members of safety 
regions (fire 

department, police 
force, medical 

troupes, office of a 
dike-reeve) 

Development of an 

instrument for self- 

assessment of 

Competencies for Network 
Centric Organizations 

obs 

3 

Card-sort 

NCOQ 

74 

military 

Navy personnel 

involved in 

Information 

Management 

Translating Lessons Learned 
into Education and Training, 
case Information 

management 

obs 

4 

Card-sort 

CSLQ 

16 

Research 

institute 

Researchers 

behavioural and 

social sciences 

Determinants for innovation 
in multi-disciplinary teams 

obs 

5 

Card-sort 

CSLQ 

22 

training and 
organization 
consultancy 
agency 

Teachers and 

Consultants 

Measuring competencies 
for self-directed learning 

obs 

6 

Card-sort 

CSLQ 

86 

crisis 

management 

studies 

Students 

Serious gaming 

intervention, effect on self- 
directed learning 

quasi 

7 

Card-sort 

CSLQ 

335 

SMEs 

Employees from 22 
SMEs in the potato, 
vegetable & fruit 
retail business 

Assessing the learning 
culture 

obs 

8 

Card-sort 

CSLQ & NCOQ 

40 

crisis 

management 

organizations 

CoPI members from 4 
different safety 

regions. 

Development of a tool to 
Improve Team Situation 
Awareness in 

multidisciplinary Crisis 

Management Teams 

exp 

9 

Card-sort 

CSLQ 

57 

Military 

Personnel from the 
Royal Netherlands 

lth-CIMIC (Civil- 

Military Cooperation) 
battalion 

Development of a 

ubiquitous learning 

environment 

obs 


Card-sort 

CSLQ 

43 

Military 

Navy personnel 

Training instructors in using 
the Job Oriented Training 
approach 

obs 

10 

Card-sort, 

Prototype 

Profile, 

CSLQ 

66 

Military 

Navy personnel 

Minimal e-coaching 

intervention to support 
social learning in an online 
Community of Practice 

exp 
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Refr. 

nr. 

iSELF modules 

used 

Questionnaire ( 1 ) 

N 

Domain 

Subjects 

characteristics 

Study aim 

Research type 
12 ) 

ii 

Card-sort, 

Prototype 

Profile, 

CSLQ 

29 

Military 

Personnel from the 
Royal Netherlands 

National Reserve 

Corps 

Minimal e-coaching 

intervention to support self- 
directed learning using e- 
mail 

exp 

13 

[regular online 
questionnaire], 
Profile, 

Feedback 

MilLead 

79 

Military 

Royal Netherlands Air 
Force, Military rank: 
Sergeant-major to 

Captain. 

Development of an 

instrument for self- 

assessment and invitee- 
assessment of leadership 
competencies 

obs 

14 

Card-sort, 

Profile, 

Feedback 

CSLQ 

11 

Crisis 

management 

organizations 

COPI and ROT 

members from 

different safety 

regions 

Usability pilot 

obs 


(1) see Table 1 for more information about the questionnaires 

(2) obs=observational, quasi= quasi-experimental, exp: experimental; [Decision rule: I: are the subjects 
randomly assigned to conditions? II: has the experimenter functional control over independent variable(s): I 
yes+ II yes = exp.; I no+ll yes = quasi.; I no+ II no = obs.] 

4.2 Phase 1: Learner-profiling 

4.2.1 The iSELF alpha version 

This version contained an internet based card-sort module and a half-automatic prototype of the profile 
module. The card-sort module was transformed in an online version, looking like a 'solitaire' web-game. The 
prototype profile module used MS Excel to produce a graph and MS Word for handmade individual reports. 

4.2.2 Context 

The alpha version was tested in three steps: (1) several try-outs of the card-sort module, (2) the profile module 
was added and presented to the learners and (3) the effect of receiving and discussing a personal profile was 
tested. To try out the card-sort module, various self-evaluation questionnaires were developed (see Table 1) 
and psychometrically tested in crisis management organizations, the military and with employees from small 
and medium enterprises (see Table 2, study 2-11 & 13). The iSELF was presented to the participants from 
within a learning management system (I LI AS, MOODLE) or with a hyperlink in an email. The data-output of the 
card-sort module was used to identify core competencies using the statistical package SPSS. 

Next, in one of these studies, the participants received a personal profile about their Self-directed learning 
competencies. A group of Navy personnel received a profile after a pre-post-test intervention study (see Table 
2, study 10). The profile showed the individual results on three points of measurement and the overall average 
scores of the whole group. Afterwards the participants were interviewed. 

Finally, in one study (see Table 2, study 11) military personnel participated in an experiment in which the 
experimental group received their profile and minimal e-coaching by email. Compared to the control group, 
their taking control for their learning increased significantly because of the intervention. 

4.2.3 Experiences 

The card-sort module of the iSELF alpha version was used successfully: the technique worked, questionnaires 
could be validated and the participants valued the module positively. The profile prototype was evaluated as 
useful in the communication with the participant. Moreover, receiving and discussing a profile in itself 
improved some aspects of self-directed learning. 
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4.3 Phase 2: Prototyping the match of learner profiles with learning solutions profiles 

4.3.1 The iSELF beta version 

This version contained all iSELF-functionalities (card-sort, profile, feedback). 

4.3.2 Context 

The beta version was tested in two steps: First, the profile and feedback modules were used as an addition to a 
'classic' online-questionnaire embedded in MOODLE. It was developed for self-assessment and invitee- 
assessment of leadership competencies in Air force personnel (see Table 2, study 12). The suggestions the 
participants received from the feedback module included formal (training, e-learning) and informal learning 
possibilities (e.g. documents, discussion, movies). The employees could choose to use the iSELF or not and 
could invite others to assess them if they wished. The next step was a usability pilot (see Table 2, study 13) with 
the complete iSELF that was carried out with employees from crisis management organizations. Afterwards 
participants were interviewed about their evaluations. 

4.3.3 Experiences 

In the Air Force case, using the iSELF was voluntary. Many employees did use it and their numbers are still 
increasing. In the usability pilot, participants were enthusiastic about the iSELF. One of the conclusions was that 
matching the learner profile with learning possibilities saves time and will improve adaptation of the learning 
environment to the learner. These experiences indicate that the tool is usable and useful. 

5. Discussion and conclusions 

This paper describes the iterative development, testing and evaluation of the iSELF: an Internet-tool for Self- 
Evaluation and Learner Feedback. The tool is designed to stimulate self-directed learning in a ubiquitous 
learning environment and our experiences so far confirm its usefulness. 

The experiences with the card-sort module in a large and diverse group of participants proved that the 
technique was highly appreciated. When they had the possibility to use it anytime and anywhere, participants 
used it voluntarily. The playful appearance, the user-friendly operation and the clear, relevant content 
increased motivation to use it. 

The profile module helps the learner to gain insight in his/her own development in relation to the 
competencies important for his/her work. These competencies always need to be identified before they can be 
used in the iSELF. We emphasize the importance to assess competencies that are specific for a group of 
learners, and at the same time generic enough to remain relevant in our rapidly changing society. This helps 
the learner to reflect on 'real life' (work)needs. 

The possibility to compare their results with previous results or with those of invited peers, colleagues, coaches 
or subject matter experts, seemed to improve reflection as intended. In future, it is possible to include e-coach 
possibilities that will stimulate reflective competencies more explicitly. 

The feedback module presents suggestions for learning, which helps the learner to manage and monitor 
his/her own learning process and to take control over educational decisions. 

An important requirement for the iSELF is that it should support learning anytime, anywhere. Some 
organizations use Learning Management Systems (LMS) to provide learning solutions like e-learning, others use 
the LMS as a portal and offer links to learning solutions outside the LMS. Therefore, the iSELF needed the 
possibility to be embedded into Learning Management Systems. The beta version was tested both inside and 
outside an LMS and could thus be used anytime and anywhere. LMS also offer the possibility to monitor the 
learner's progress. When the iSELF is used within an LMS, it is possible to make the learner profiles available 
for monitoring. 

In spite of these positive findings, it is clear that there are some limitations and many challenges left in our 
quest to provide learners with a self-assessment and feedback tool for self-directed ubiquitous learning. For 
instance, it is technically possible to use the iSELF for selection purposes by combining the job-requirements 
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with the individual competencies. However, to support learner control and to provide a 'safe' learning 
environment, it is important that the learner is the only one who can see personal evaluation results, until 
he/she decides otherwise. Therefore, we use this tool exclusively for learning or career suggestions and not for 
selection. It is, however, possible to combine the individual results and present them an average on group- 
level, for organizational purposes. 

As described in the introduction, ubiquitous learning is a way of learning in which learners follow their own 
trails of interest, scaffolded by coaches, peers and tools for thinking and learning (Dieterle & Dede 2007)□. It 
includes on- and off-line, formal and informal learning. To support all that, the iSELF should be available 
through internet independent of the learning solution chosen. iSELF was not tested for off-line learning 
solutions and therefore we do not know if learners who prefer off-line solutions will use the on-line iSELF. 
Future research should look into this limitation. However, the other way around does not present any 
problems: the on-line iSELF can refer to off-line learning solutions. The profile of any learning solution that is 
evaluated by educational experts can be included. As such the iSELF can be used for both formal and informal 
learning. 

Another challenge that influences the usefulness of the iSELF is that in a self-directed ubiquitous learning 
environment there is no pre-defined learning content and learners can select content from on- and off-line, 
formal and informal learning (Gutl et al. 2011). Especially informal learning can help learners to collaborate 
with others when learning, a requirement for self-directed learning. Of course, it is almost impossible to 
provide profiles of all possible learning solutions. To start with, the most useful or important learning 
possibilities for a certain group of learners can be profiled. In addition, it might be possible to ask learners for 
suggestions for useful learning solutions, which in itself increases collaborative learning. 

When judging the match of learning solutions with the learners profile, a personalisation of learning solutions 
is needed to knowledge level, goals and other characteristics of individual learners (Papanikolaou et al. 2002). 
For adaptation, one has to consider both the content and the didactics of a learning solution. There are 
interesting developments concerning content adaptation to the learners profile: one option is to filter out 
unsuitable course materials to reduce cognitive load (Barker 2011). To do so, for instance the Automatic test 
item creation can be used (Gutl et al. 2011), or content selection based on item response theory (C. M. Chen et 
al. 2005). However, one has to bear in mind that for self-directed learners, technology must not take away 
control from the learner, but in stead provide stimuli to increase competencies for self-directed learning. In 
that respect didactical adaptation to the learner's profile is needed. Coaching and instruction style must fit the 
learners need for personalized learning guidance. In our study with minimal e-coaching, it appeared that 
receiving and discussing a profile in itself improved some aspects of self-directed learning. There are 
interesting developments concerning the adaptation of the didactical approach to the learner's profile as well: 
for instance in the development of Intelligent Support for Discovery Learning (Veermans, De Jong, & Van 
Joolingen, 2000). Veermans et al (2000) showed that it is possible to develop algorithms that provide the 
learner with advice, while preserving the open nature of the discovery environment. 

A limitation of the presented work is that the development in phases was performed using many different 
studies and convenience samples. Some of the development iterations would have been different if a 
dedicated science program with sharply defined samples could have been used. On the other hand, the large 
amount of studies in different domains, with different sets of competencies and with different study designs, 
provided many challenges that helped us to develop a better tool. As a result the iSELF was developed to be 
not only a learner tool but a scientific tool as well. Extensive analyses could be made using the output of the 
card-sort. Plus-point of the card-sort was the absence of missing data and a good response. It facilitated 
questionnaires 'under construction' so one can keep up with new developments and new competencies 
needed in the workplace. 

A flexible and innovative economy requires permanent adaptations of knowledge, skills and attitudes. Formal, 
initial training alone cannot meet the need for the development of working individuals to face these 
challenges. There is, therefore, a growing need for self-directed, flexible and innovative employees who can 
and will keep on learning throughout their entire lifespan. Research had shown that fostering students to 
become self-regulated learners is complicated and should be seen as a long-term process (Van den Boom et al. 
2007). The iSELF will stimulate self-directed learning in a ubiquitous learning environment and will help to 
create learners for life. 
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